Image fusion system and method

ABSTRACT

A contrast-based image fusion system and method of processing multiple images to form a processed or fused image including regions selected from one or more images. Images are divided into image regions. Portions of the images are filtered if necessary. A contrast map is generated for each image via a convolution kernel resulting in a contrast map with contrast values for each image region. Contrast values are compared and image regions are selected based on a selection criteria or process such as greater or maximum contrast. The selected image regions form the fused image. If necessary, the luminance of one or more portions of the fused image is adjusted. One sensor is selected as a reference sensor, and an average intensity of each region of the reference sensor image is determined across the reference sensor image. The intensity of one or more regions in the final image is adjusted by combining the determined average luminance values and intensity values of the final image.

FIELD OF THE INVENTION

The present invention relates generally to imaging systems and methods,and more particularly, to an imaging system and method that selectivelyfuse or combine regions of images from two or more sensors to form asingle, processed image.

DESCRIPTION OF RELATED ART

Image fusion generally refers to combining or merging portions of two ormore images into a single processed image. Image fusion is commonly usedwhen two or more detectors are used in generating an image, whereby theimage displayed to a user or provided to an automated processing systemis combined from information provided by each of the sensors.

One manner in which known systems combine images from different sensorsis by merely adding the two images together on a pixel by pixel basis.Thus, for example, for rendering a two-dimensional (2-D) processed imageof pixels arranged in an n×m matrix wherein each pixel position isidentified by the position (x,y), a value or data in pixel (1,1) of thefirst image is added to the data or value in pixel (1,1) in the secondimage, a value or data in pixel (1,2) of the first image is added to thevalue or data in pixel (1,2) of the second image, and so on for eachpixel through pixel (n,m) of both images. Other known systems perform avariant of this technique and calculate the average of the values ineach pixel instead of adding the two values. Thus, the final imagecontains averaged pixel values.

These systems, however, have a number of shortcomings. First, knownimage fusion techniques typically result in undesirable and unnecessarydistortion. For example, if a portion of an image is clear andunderstandable by a user, while the corresponding portion of a secondimage is blurry, then adding or averaging pixel values can distort theclear image into one that is less clear. This undesirable effect is theresult of incorporating elements of the blurry pixel(s) into the clearpixel(s) through addition or averaging. As a further example, addingunnecessary background regions to a bright image region can decrease thecontrast and quality of the bright image region. For example, if regionsof two images have high dominance or are bright, then adding two brightregions together can result in a final image that is “overexposed” ortoo bright. This results in a saturated image. Finally, averaging twodim image regions can result in a relatively dim image, and imageregions that were originally dim can have their brightness furtherreduced.

Other known systems have attempted to overcome these shortcomings usingtechniques that identify patterns in images and forming a fused image onthe basis of patterns. Each source or original image is decomposed intomultiple, lower resolution images using filters with differentbandwidths (e.g., based on Gaussian roll-off or a Laplacian “pyramid”approach). The pyramid approach is based on using different resolutionsfor different image regions—coarse features are analyzed at lowresolution, and fine features are analyzed at high resolution. Thesesystems, however, are also deficient in that the complete image fromeach sensor is received before the process of constructing a pyramid canbegin. This requirement typically results in a time lag of at least oneimage from the slowest sensor. Such a time lag is unacceptable insensors placed on fast moving platforms, such as aircraft or othervehicles, or more generally where real-time operation is desired.

Other known systems use a technique in which the Laplacian method ismodified and source images are decomposed into patterns which areassigned saliency values or weights. A pattern is “salient” if itcarries information that is useful to understanding the image. A finalimage is formed on the basis of “weighted” patterns. These techniques,however, can also be deficient in that they typically involve analyzingand assigning saliency weights to each pixel or region of the entireimage. Then, the entire image is processed. Thereafter, the salientpatterns are selected. As a result, excessive time is wasted analyzingregions of entire images and their corresponding saliency values.

These shortcomings are particularly problematic when known image systemsare used in connection with time sensitive activities, e.g., landing anairplane, driving a tank, etc. In these situations, it is desirable thatclear images be generated quickly. Known techniques, however, typicallycannot generate quality images within these time constraints ortypically do so only after full images are available for processing.

Accordingly, a need exists for a method and system that effectively andefficiently select useful, pertinent or relevant information from sourceimages to form a more informative or useful processed image whichincludes relevant, pertinent and useful information from each of thesource images in a time efficient manner. Further, it is desirable toapply the selective image fusion technique to a variety of detectors orimage generators to provide flexibility for use in differentapplications.

SUMMARY OF THE INVENTION

The present invention provides a method and system for selectivelycombining regions of images generated by different sensors (also hereinreferred to as sensor or source images) to form a processed or fusedimage using the relevant information from the sensor images. The methodand system are implemented by dividing each sensor image into imageregions, and generating for each image region a map of contrast valuesby means of for example, a convolution. The map of contrast values forone sensor image is then compared to the corresponding map of contrastvalues for the other sensor image. Between or among the comparedcontrast values, one contrast value is selected based on a selectioncriterion, which can be, for example, the greater of the two or morecontrast values compared. The image regions corresponding to theselected contrast values are then used to form the processed image.According to the present invention the image regions can be divided on apixel-by-pixel basis, based on groups of pixels, or based on arbitrarilyshaped regions.

In yet further accordance with the invention, each sensor detects adifferent wavelength. Also in accordance with the present invention,images from different types, numbers, and combinations of sensors can beprocessed. Sensors that can be used include infrared (IR),radio-frequency sensors (e.g., active sensors such as radar, or passivesensors such as radiometers)”.

In still further accordance with the present invention, image regionsfrom a plurality of sensors are combined to form the processed image.

In further accordance with the present invention, contrast maps forimages from a first sensor and a second sensor are combined to form anintermediate contrast map, which is then compared with a contrast map ofthird image to form the processed image.

In further accordance with the invention, the image fusion method andsystem are used in connection with directing a moving vehicle such as anaircraft, watercraft, automobile, or train.

In further accordance with the invention, the intensity or luminance ofone or more image sections is adjusted across the processed image. Onesensor is selected as a reference sensor, and an average intensity ofregions of the reference sensor image is determined. The intensity ofthe same or corresponding region or an adjacent region in the processedimage is adjusted by combining the determined average luminance valuesof the reference image and intensity values of the processed image.

Also in accordance with the invention, the method and system areimplemented to filter portions of the sensor images before contrastcomparisons are performed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an embodiment of a system in accordance with thepresent invention, including a processor or computer, two sensors, and adisplay within a moving vehicle, such as an aircraft;

FIG. 2 is a flow diagram illustrating the processing of images generatedby sensors to form a processed or fused image;

FIG. 3 is a flow diagram illustrating the manner in which contrastvalues are compared;

FIG. 4 is a flow diagram illustrating the manner in which luminance of aprocessed image is adjusted;

FIGS. 5A-C are black and white photographs illustrating respectiveimages of radar sensor, an infrared (IR) sensor, and a processed imageincluding regions selected from the radar and IR images based on aselection process or criteria;

FIGS. 6A-F illustrate dividing an image into different image regions,including on a pixel-by-pixel basis, groups of pixels, or arbitrarilydefined regions;

FIGS. 7A-B are black and white photographs illustrating contrast mapsthat are generated for each image;

FIGS. 8A-B are black and white photographs illustrating contrast valuesselected from the contrast maps of FIGS. 7A-B based on a selectioncriteria;

FIG. 9 is a flow diagram illustrating the processing of a plurality ofimages by comparing all of the contrast values of the images to form aprocessed or fused image;

FIG. 10 is a flow diagram illustrating the processing of a plurality ofimages by performing multiple comparisons of contrast values to form aprocessed or fused image;

FIGS. 11A-B are black and white photographs illustrating a processed orfused image before and after luminance correction;

FIGS. 12A-B are black and white photographs generally illustratingspatial filters;

FIGS. 13A-B illustrate filter plots for a radar and IR sensor,respectively;

FIGS. 14A-F are black and white photographs illustrating radar and IRimages, the filter function or effect, and the filter function appliedto the radar and IR images; and

FIGS. 15A-E are black and white photographs illustrating a comparison ofweighting functions with and without a roll effect.

DETAILED DESCRIPTION

In the following description of embodiments of the invention, referenceis made to the accompanying drawings which form a part hereof, and whichis shown by way of illustration specific embodiments in which theinvention may be practiced. It is to be understood that otherembodiments may be utilized as structural changes may be made withoutdeparting from the scope of the present invention.

With reference to FIG. 1, a view from a cockpit in an aircraft, a systemS of the present invention is shown, having sensors 100, 102 and aprocessor 110 and a display 120. The sensors 100, 102 provide respectiveimage data or streams 104, 106 (i.e., sensor or source images) to theprocessor 110, e.g., a computer, micro-controller, or other controlelement or system. The sensors can detect the same, overlapping, ordifferent wavelengths. Moreover, the sensors can also detect the samefield of view, or overlapping fields of view.

The processor 110 is programmed to selectively combine regions from eachimage 104, 106 into a processed or fused image 115. More specifically,the processor 110 compares regions of each image 104, 106, and selectsimage regions based on a selection criterion, for example, a comparisonof contrast values representing the apparent difference in brightnessbetween light and dark areas of sensor images. The processor can beprogrammed to consider different selection criteria including, but notlimited to, the greater or maximum contrast values of each comparison.Thus, the processing system essentially extracts the desirable regionsor regions of choice based on the selection criterion from one or moreor all of the images. The selected regions are pieced together to formthe fused image 115 (much like a jigsaw puzzle is formed from multiplepieces, except that each piece of the puzzle can be selected frommultiple sources). The “puzzle pieces” or image regions can come from asingle image, some of the images, or all of the images. The fused image115 is then presented to the pilot or user through the visual display120. The fused image can also be provided to an image processor orcomputer for further processing.

While FIG. 1 illustrates the application of the system S in an aircraft,those skilled in the art will recognize that the system can be appliedto many other vehicles and used in various applications as will bedescribed.

The technique of fusing or selectively combining portions of images 104,106 into a processed image 115 is illustrated in the flow diagrams ofFIGS. 2-4. As shown in FIG. 2, in step 200, each sensor generates animage, and the image data is provided to the processor. In step 202, ifdesirable, image regions can be filtered for exclusion from processing,exclusion from the processed image or to de-emphasize their contributionto the processed image. In step 204, contrast values of correspondingregions of each sensor image are compared. In step 206, the selectioncriterion is applied for selecting or identifying certain contrastvalues. In an embodiment of the system S, the selection criterion may beto select or identify the greater or maximum contrast values; however,the selection criterion, criteria or process may be altogether differentin another embodiment of the system S depending on how the system isutilized. In step 208, image regions corresponding to the selected oridentified contrast values are identified or selected. In step 210, theselected image regions are combined, that is, effectively “piecedtogether” to form the fused or processed image. Then, in step 212, ifdesirable, the intensity or luminance of the processed or fused image isadjusted or corrected to produce a clearer image.

FIG. 3 further illustrates step 204 or comparing contrast values. Instep 300, each sensor image is divided into a plurality of imageregions. Then, in step 302, a contrast map for each sensor image isgenerated. Each contrast map includes contrast values for each definedimage region. In step 304, contrast values of image regions of onesensor image are compared to contrast values of corresponding imageregions of the other sensor image(s). Corresponding image regions asused in this context refers to sensor images that at least overlap. Forexample, if the field of view of one sensor image includes an airfieldrunway, this sensor image “overlaps” with the field of view of anothersensor image if the latter also includes the same airfield runway. Ifthe fields of view of the two sensor images are identical (or nearlyidentical) with each other, the images are deemed to have 100% percentoverlap (so on and so forth).

Turning now to FIG. 4, step 212 or adjusting the intensity or luminanceof the fused image, is illustrated in further detail. In step 400, onesensor is selected as a reference sensor, i.e., the sensor for whichluminance values are to be matched. Then, in step 402, the averageluminance or intensity of image regions of the reference sensor image(e.g. cross-sectional lines) is determined across the image. Next, instep 404, the intensity of one or more regions of the fused or processedimage is adjusted by combining the determined average luminance valuesand intensity values of the fused image to form a luminance-correctedfused image. The intensity adjustment can be applied to the same regionor an adjacent to following regions. For example, the adjustment can beapplied to the same region or line 406 for which the intensity wasdetermined, or an adjacent or following region or line 408 in the fusedimage.

Those persons of skill in the art will recognize that the image fusionmethod and system can be used in many different environments andapplications that process multiple images. For example, besides anaircraft (e.g. an airplane, jet, helicopter, etc.) the method and systemcan be implemented in other moving vehicles such as a watercraft, anautomobile, or a train. Moreover, the image fusion method and system canbe used to display images from medical instruments (which use, e.g.,ultrasound, infrared, laser imaging or tomography sensors), andsurveillance systems. Indeed, many applications can benefit from theselective fusion of image regions to form a processed or fused imagethat includes relevant information or information of choice from eachsensor image.

However, for purposes of explanation, this specification primarilyrefers to images related to an aircraft. Such images may be related tolanding, taxiing, takeoff, or cruising of the aircraft and in connectionwith applications to prevent Controlled Flight Into Terrain (CFIT). As aspecific example of how the system can be used in aircraft applications,this specification refers to processing images generated by a radarsensor and an IR sensor. However, as will be explained, many differenttypes, numbers, and combinations of sensors and sensor images can beprocessed. Accordingly, the example system and method explained in thisspecification can be used with many different applications.

Images and Sensors

Turning now to FIGS. 5A-C, sensors 100, 102 generate respective images104, 106, e.g., images 500, 510 illustrated in FIGS. 5A-B. Selectedregions of one or both images are used, that is, effectively joined orpieced together to form a fused or processed image 115, e.g., the fusedimage 520 illustrated in FIG. 5C. Depending on the content of the sourceimages, it may be desirable to further process the fused image, e.g., aslater explained in connection with FIGS. 11A-B.

More specifically, FIG. 5A illustrates an image 500 of a runwaygenerated by an infrared (IR) sensor. The IR sensor can operate atvarious IR wavelength ranges, e.g., 0.8 to 2 μm, 3-5 μm, 8-12 μm, orcombinations and extensions thereof. One example source of an IR sensorthat can be used is available from BAE SYSTEMS, Infrared ImagingSystems, Lexington, Mass. FIG. 5B illustrates the same runway in thesame or nearly the same runway scene, but as image 510 generated by aradar sensor. Radar sensors can be X, K, Ka or other band radar sensors.Suitable radar sensors for use with the present invention are availablefrom, for example, BAE SYSTEMS Aircraft Controls, Santa Monica, Calif.

In this instance both the IR sensor and the radar sensor generallyprovide the same or overlapping fields of view such that objects orconditions visible in both fields of view may be better detected by onesensor than the other sensor. Those of ordinary skill in the art willrecognize that the system and method can be applied to images withdifferent degrees of overlap or fields of view, as later described.Moreover, while the described embodiment provides a specific example ofa system including radar and IR sensors and images, different types,numbers, and combinations of sensors and images can be utilized. Forexample, the system can also be used with ultraviolet (UV) sensors, oneexample UV sensor being available from Pulnix America, Inc., Sunnyvale,Calif. Further, one of the sensors can be based on an active or passiveradio-frequency (RF) system such as an imaging radar or radiometer,operating in various RF bands including but not limited to 10, 35, 76,94, and 220 GHz, one example of such a sensor being available from TRW,Inc., Redondo Beach, Calif. As a further example, a sensor can be anultrasonic sensor, such as those ultrasonic sensors utilized in medicalimaging applications available from General Electric Medical SystemsDivision, Waukesha, Wis. A sensor can also be a visible band sensor,e.g., a low-light level visible band sensor, Charged Coupled Device(CCD), or color or grayscale camera which can use natural or artificialillumination, available from Panasonic, Inc., Secaucus, N.J.

Further, the image fusion system can be configured to process imagesfrom a plurality of sensors, e.g., three, four, or other numbers ofsensors. One possible combination of sensors includes two IR sensors anda radar sensor. The images from all of the images can be jointlyprocessed and selectively combined into a processed image. For example,images A, B, and C can be selectively combined into processed or fusedimage D. Alternatively, two sensor images can be processed, the resultof which is processed with a third sensor image to form a processed orfused image or its representative contrast map. For example, images Aand B are combined into image C or an intermediate contrast map C thatis subsequently selectively combined with image D or contrast map D toform fused image E or further intermediate contrast map, and so on,until all of the images are processed to form a fused image. Indeed,different combinations of different number of sensor images can beprocessed with different iterations of comparisons as desired or needed.

The selection of the type of the sensors may depend on the conditionsand environment in which the sensor is used. As previously discussed,one type of sensor may be better suited for one environment, whereasanother sensor may be better suited for a different environment. Morespecifically, certain types of sensors may provide clearer imagesdepending on whether the environment is daylight, night, fog, rain, etc.and depending on whether the image is distant or near. For example,radar sensors typically provide better images in fog conditions comparedto IR sensors, but may lack the photograph-like qualities of IR images.

Comparing Contrast Values of Image Regions

Image region contrast values are compared (step 204) by dividing imagesinto regions, generating contrast maps based on the defined regions, andcomparing the corresponding contrast map values using a selectioncriterion or criteria. The comparison is based on aligned orpre-registered images or images arranged to permit comparison of relatedimage regions. Thus, if images that do not overlap are processed, theyare pre-registered or aligned such that related regions are compared asdescribed in further detail below. Contrast values are then selected(step 206), for example, on a selection criterion favoring the greateror maximum contrast values. Other selection criteria may also beutilized, for example, temporal persistence, brightness, color, etc.

Dividing Images Into Regions

Initially, sensor sensors are divided into sensor regions as illustratedin FIGS. 6A-F. Images can be divided on a pixel-by-pixel basis 600a-b,601a-b (FIGS. 6A-B) or based on groups of pixels 602a-b, 604a-b (FIGS.6C-D). A pixel or group of pixels can be “black or white” to represent amonochrome image, different shades of gray (gray scale) to represent animage with different levels of intensities. A pixel or group of pixelscan also have red, green, and blue dots which are activated to form partof a color image. Further, image regions can be defined as havingarbitrary shaped regions or boundaries 606a-b, 608a-b, 610a-b, 612a-b(FIGS. 6E-F). As a result, one image region can be compared to anothercorresponding image region, for each region in each sensor image. Forexample, referring to FIGS. 6A-B, region 600a (x₁=1, y₁=12) can becompared to region 600b (x₂=1, y₂=12); and region 601a (x₁ =17, y ₁=10)can be compared to region 601b (x₂=17, y₁=10) can be compared to region601b (x₂=17, y₂=10).

For purposes of explanation, FIGS. 5A-B and the related example imageregions illustrated in FIGS. 6A-F involve the same or essentially thesame images with generally aligned or pre-registered image regions,e.g., aligned or pre-registered pixels, groups of pixels, or arbitraryshaped regions. In other words, FIGS. 5A-B illustrate overlapping images(100% overlap) or images having a high degree of overlap (almost thesame sensor images). As a result, the image regions in FIGS. 6A-F arealigned with each other in a series of corresponding image regions.Thus, an object (e.g., tree 607) is in nearly the same relative positionwithin the sensor images, residing in the identical image regions ofboth sensor images, regardless of how the sensor images are divided intoimage regions.

However, those skilled in the art will recognize that the system andmethod can be utilized with different numbers, types, and combinationsof sensor images having different degrees of overlap depending on thelocation, position, field of view, and detection capabilities of asensor. In cases involving different degrees of overlap, the imageregions can be aligned or pre-registered such that the comparisons canbe performed.

For example, sensors can be positioned closely together (e.g., near thefront or bottom of the aircraft) to detect essentially the same images,such as the runway scene illustrated in FIGS. 5A-B. As a result, theimage regions in the same or similar images are generally aligned witheach other in a corresponding manner, as illustrated in FIGS. 6A-F. Inthese cases, the image regions to which a selection process or criteriais applied (or image regions “competing” for selection and use informing the processed image), can be considered to be all of the alignedimage regions in FIGS. 6A-F since the images are generally the same withthe same boundaries and fields of view.

As a further example, one sensor may detect a first image whereas adifferent sensor may detect most of the first image, but additionalscene elements as well. This may occur when, for example, sensors arepositioned apart from each other or are positioned to have differentfields of view. In this instance, the selection process may be appliedto some or all of the overlapping regions. The image regions areprocessed by application of a selection process or criteria such ascontrast comparisons. The competing regions are compared, and the imageregions are selected to form the processed or fused image. The imageregions that are not overlapping or are not competing can be processedin different ways depending on, e.g., the quality of the source andfused or processed images, the types of sensors, and user and systemneeds. For example, non-overlapping images can be added to the processedimage as filler or background. Alternatively, non-overlapping regionscan be discarded and precluded from inclusion in the processed for fusedimage. In some cases, the overlapping regions may not be processeddepending on the particular system and application.

Thus, the method and system can be utilized with images having differentdegrees of overlap and image regions having different degrees ofalignment. The overlap and alignment variations may result from sensorshaving different detection capabilities and positions. However, forpurposes of explanation, this specification and supporting figures referto and illustrate images having a high degree of overlap with aligned,corresponding image regions. As a result, most or all of the imageregions are competing image regions and processed with the selectioncriterion. However, the method and system can be configured to processother image region configurations having different degrees of overlap,alignment, and correspondence.

Generating Contrast Maps

As shown in FIGS. 7A-B, contrast maps 700, 710 are generated forrespective radar and IR images. Each contrast map includes a contrastvalue for each defined image region within that contrast map. Continuingwith the example using radar and IR sensors, FIG. 7A illustrates acontrast map 700 for the radar image, including contrast values, one foreach of the image regions into which the radar image has been divided.Similarly, FIG. 7B illustrates a contrast map 710 for the IR image,including contrast values, one for each of the image regions into whichthe IR image has been divided. In accordance with the present invention,there may be any number of image regions in each contrast map 700 and710, where such number should preferably be equal and the image regionscorresponding where the radar and IR sensors provide 100% overlappingimages.

For this example radar map, the contrast values in the general top andbottom portions 702, 706 of the image/map are of a relatively lowervalue, and the contrast values in the general middle portion 704 are ofa relatively higher value. For the example IR map, the contrast valuesin the general middle portion 714 are of a relatively lower value andthe contrast values in the general top and bottom portions 712, 716 arerelatively higher in value.

In accordance with the present invention, contrast maps includingcontrast values for each image region are generated via, e.g., aconvolution with an appropriate kernel. One example convolution andkernel that can be utilized is a 2-dimensional (3×3) normalizedconvolution kernel:K_(c)*S1(x,y),K_(c)*S2(x,y)wheredenotes a convolution; $K_{c} = {{\begin{matrix}{- \frac{1}{2\sqrt{2}}} & {- \frac{1}{2}} & {- \frac{1}{2\sqrt{2}}} \\{- \frac{1}{2}} & {2\frac{1 + \sqrt{2}}{\sqrt{2}}} & {- \frac{1}{2}} \\{- \frac{1}{2\sqrt{2}}} & {- \frac{1}{2}} & {- \frac{1}{2\sqrt{2}}}\end{matrix}} \cong {\begin{matrix}{- 0.354} & {- 0.500} & {- 0.354} \\{- 0.500} & 3.414 & {- 0.500} \\{- 0.354} & {- 0.500} & {- 0.354}\end{matrix}}}$x,y are spatial coordinates of the image, ranging from 0 to the imagewidth (w) and height (h), respectively;

-   S1 is the first sensor image, e.g., a mmW radar image stream; and-   S2 is the second sensor image, e.g., an IR image stream, assumed    spatially pre-registered to or aligned with the first or radar    image.

The example kernel K_(c) includes values that reflect a distance metricfrom its center. A contrast map is generated including contrast valuesfor each image region of each image as a result of the convolution.

The processor can execute the convolution with a program in C-code oranother programming language, or in dedicated integrated circuithardware. Real-time implementation of the convolution can be achievedthrough the use of a Digital Signal Processor (DSP), Field ProgrammableGate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs) orother hardware-based means.

Selection of Contrast Values

FIGS. 8A-B illustrate the pixel values that are used in forming theprocessed image, as selected based on a selection criterion performed onthe comparison of contrast values in the contrast maps of FIGS. 7A-B. Inthis example where the selection criterion operates to select thegreater of contrast values between an image region of the radar imageand a corresponding image region of the IR image, FIG. 8A illustratesthe pixel values of the selected (radar) contrast values of FIG. 7A,which as mentioned above, reside generally in the middle portion 800 ofthe radar image. Similarly, with the system S operating under the sameselection criterion, FIG. 8B illustrates the pixel values of theselected (IR) contrast values of FIG. 7B, which as mentioned above,reside generally in the top and bottom portions 810 and 820 of the IRimage.

Each image region associated with a selected contrast value is selectedfrom each image and then combined (or “pieced together”) with other suchselected image regions to form the processed or fused image, e.g., thefused image illustrated in FIG. 5C. Thus, in this example, the criteriafor selecting image regions based on maximum contrast values can bestated as follows:F_(max-con)(x,y)=max{K_(c)*S1(x,y), K_(c)*S2(x,y)}where the “maximum criteria” operation is performed or an arbitraryshaped region. Thus, the selection of image regions based on maximumcontrast essentially serves as a pixel value resulting in a fused imagethat includes a combination or superset of image regions from differentimages. The image regions may, as a result of the selection process, beselected all from a single image or from multiple images depending onthe content and contrast values of the images. Some sensor images maynot contribute any image regions to the fused or processed image. Forexample, if a first image has all of its contrast values identified orselected, then the processed or fused image will be the same as thefirst image. As a further example, if contrast values are selected fromsecond and third images but not the first image, then the fused imageincludes regions from the second and third images but not the firstimage. Thus, in the processed image having image regions A, B and C,image region A may be from sensor image 1, image region B may fromsensor image 2 and image region C may be from sensor image 3.

The previously described example involving the application of aconvolution results in the generation of two contrast maps. Indeed,other numbers and combinations of convolutions can be performed togenerate multiple contrast maps for use in multiple comparisons ormultiple sensor images. For example, referring to FIG. 9, images 900-902are generated by respective sensors. A convolution with an appropriatekernel 910-912 is applied to the data of respective images 900-902 togenerate respective contrast maps 920-922 as follows:K_(c)*S1 (x,y), K_(c)*S2 (x,y), K_(c)*S3 (x,y)where the third sensor S3 is also an IR sensor, for example. Thosepersons of ordinary skill in the art will recognize that differentkernels can be used with the same or different sensors. Thus, a processinvolving three convolutions can use, for example three differentconvolution kernels.

Then, corresponding contrast values of the three images are compared930, and contrast values are selected 940 based on a selectioncriterion. The image regions selected 945 correspond to the selectedcontrast values. The selection of image regions based on maximumcontrast value criteria can be expressed as follows:F_(max-con)(x,y)=max{K_(c)*S1(x,y), K_(c)*S2 (x,y), K_(c)*S3 (x,y)}

The selected regions from one or more of the sensor images are thenpieced together to form a processed or fused image 950. Thus, in thisexample, all of the corresponding contrast values are compared together(three contrast values compared at the same time) to select the imageregion(s) having the maximum contrast value.

In an alternative embodiment, multiple iterations of convolutions can beperformed to generate respective contrast maps, the values of which arecompared in iterations to eventually form a processed image. Forexample, referring to FIG. 10, contrast maps 920-922 are formed for eachimage 900-902, as previously described via a convolution and appropriatekernel 910-912. However, instead of comparing all of the correspondingvalues of each contrast map together, iterations of contrast mapcomparisons are performed, possibly utilizing differentcontrast-selection kernels.

Thus, for example, a comparison 100 of contrast values in contrast maps920 and 921 is performed resulting in a selection of a set of contrastvalues 1010 based on, e.g., greater or maximum contrast. The selectedcontrast values are selectively combined to form an intermediate imageor contrast map 1030.

Contrast values in contrast map 1030 are then compared 1040 to contrastvalues in contrast map 922 from the third image 902. The contrast valuesare selected or identified 1050, and the image regions corresponding tothe selected contrast values are selected 1055. The selected regionsform the processed or fusedimage 1060. Those skilled in the art willrecognize that different numbers of iterations or comparisons ofdifferent numbers of contrast maps can be performed with the same ordifferent convolution kernel. Thus, the present image processing systemand method provide flexible image fusion that is adaptable to differentapplications using convolution.

Correcting Luminance of Fused Image

The luminance or brightness of the fused image can be corrected oradjusted, if desirable, depending on the types of sensors utilized andthe quality of the resulting sensor and fused images. Luminancecorrection is particularly useful when the fused image is notsufficiently clear to the pilot.

In the example involving radar and IR images, there may be noticeableartifacts in the fused image, as shown in FIG. 5C. The artifacts resultfrom the brightness or luminance of the fused image being inconsistent,resulting in discontinuous luminance across the fused image. In thisparticular example, high-contrast regions selected from the radar image(central horizontal band in this example) are generally darker relativeto the high-contrast regions from the IR image. The luminancedistribution of the resulting processed or fused image varies betweenthe luminance of the two input sensors. For example, the darker bandacross the center of the image is generally selected from the radarimage, which, in that region, has higher contrast, but lower luminancethan the IR image. This reduces the overall clarity of the fused image.

The luminance distribution within the fused image can be adjusted togenerate a clearer fused image. Luminance adjustment is performed bydetermining average luminance values in regions of an image generated bya reference sensor, and adjusting the luminance of regions of the fusedimage based on the corresponding determined values. In the exampleimages of FIGS. 5A and 5B, the luminance adjustment technique is basedon luminance typically varying in a vertical cross-section of a sensorimage (e.g., sky through horizon to foreground), but not as predictablyin any horizontal cross-section (e.g., across the image at anyparticular elevation angle).

Reference Sensor

Luminance correction can be performed by selecting one sensor as areference sensor and adjusting the luminance of the fused image to matchor approximate the luminance distribution of the reference sensor. Thereference sensor can be arbitrarily selected or based on the expectedutility of a sensor in a particular situation. For example, a radarsensor generally provides more image detail in low-visibility conditionsthan an IR sensor. However, an IR sensor typically provides a morenatural or photographic image, at least at close range.

For purposes of explanation, this specification describes the IR sensorI(x,y) as the reference sensor for luminance distribution, to capturethe natural-looking characteristics of images from that sensor. However,the radar sensor or other sensors can be the reference sensor.

Determining Average Luminance

Adjusting luminance involves determining the average intensity in thereference sensor in specific image regions, such as, for example, stripsalong each image cross-section parallel the scene horizon. The scenehorizon refers to the “actual” real-world horizon. The scene horizon maybe at an angle relative to the image horizontal during a roll, bank orother motion of an aircraft.

The average luminance of each such strip of the reference sensor imageis determined. Then, luminance values obtained from the determinationare added to each corresponding strip of the fused image to adjust theluminance of the fused image. Further, if necessary, the degree ofluminance can be weighted for a particular luminance adjustment effect.The weight λ can be used to reduce the effect of the luminancecompensation, although a value of λ=1 has been determined to provide asufficiently clear adjusted fused image in most situations.

Thus, the manner in which image is adjusted in a fused image can beexpressed as follows:${F_{LC}( {x,y} )} = {{F( {x,y} )} + {\frac{\lambda}{w}{\sum\limits_{x = 0}^{sum}\quad{l( {x,y} )}}}}$where

-   F(x,y) are luminance values of the fused image;-   λ is a weighting factor for different degrees of luminance    adjustment;-   w is the width of the image from x=0 to x=w; and-   F_(LC) (x,y) is the luminance-compensated fused image.

Those persons of ordinary skill in the art will recognize that thereference sensor image can be sampled along different cross sectionsbesides a horizontal cross section, and with different segments besidesa strip across the image. The selection of the cross section andsampling segment may depend on various factors, including the types ofsensors, sensor images, orientation of images, and application of thesystem or method. However, for purposes of explanation, thisspecification refers to cross-sectional sampling of strips of thereference sensor image, and correcting corresponding strips in theprocessed image.

An example of applying luminance adjustment is illustrated in FIG. 11.The runway scene portrayed in the fused image 1100 before luminancecorrection includes a number of artifacts that distort the processed orfused image. As a result, the runway scene is somewhat unclear,particularly in the middle portion of the image. The image 1110represents the same image 1100 after luminance correction and selectingthe IR sensor as the reference sensor.

As can be seen by comparing images 1100 (before luminance correction)and 1110 (after luminance correction), the luminance compensated imagedemonstrates less striking luminance variations in elevation, whichotherwise tend to produce a noisy image. The result is a clearer,processed or fused image.

Luminance correction of the fused image can be performed by correctingdifferent strips or regions of the fused image. For example, the meanluminance of the reference sensor is determined for an image line orstrip in the reference sensor image. The determined mean luminance valuefrom the reference sensor image is processed with, e.g. the previouslystated luminance adjustment expression, to add it to each pixel in thecorresponding fused image line or strip.

In an alternative embodiment, processing efficiency can be increased byusing the mean or determined luminance value from one line of thereference sensor image and applying it as a correction to a line in theprocessed or fused image that is adjacent to a line in the fused imagecorresponding to the determined line in the reference sensor image(e.g., the next line above or below the corresponding determined line).Applying luminance values to the following line is generally acceptablesince the mean typically does not substantially vary between successiveimage lines. However, this technique can be applied to adjust the nextline above or below the subject line, or a number of lines separatedfrom the reference line depending on luminance variation.

Luminance correction can also be adapted to situations in which thescene horizon is not parallel to the image horizontal, e.g., when anaircraft rolls or banks to one side. In this case, the scene horizontalangle and elevation are generally known from aircraft orientationsensors. Luminance correction can be calculated from the referencesensor, stored as a two-dimensional lookup table. The correctionobtained from the lookup table is applied on a pixel-by-pixel basis tothe fused image. In order to minimize latency and processing time, tablevalues can be applied to the current frame based on values calculatedduring the previous frame, if sufficient memory storage resources forthe full-image lookup table are available. These requirements can beapproximately equal to the image frame size, for example, 320×240 bytesfor an 8-bit per pixel sensor or other sizes depending on the details ofthe image produced by each sensor.

Spatial Pre-Filtering of Sensor Images

Regions or portions or sensor images can also be filtered to simplifyprocessing of comparing contrast values and application of the selectioncriteria. The filtered regions can be represented as a number less thanone to de-emphasize their contribution to the fused image, or a zero toremove them from contributing at all of the fused image, to simplify andreduce processing time.

Image regions that can be filtered include portions of the images thatwill not be included in the fused image, e.g., regions above a radarhorizon in the case of a radar sensor. If a radar sensor is utilized,there is typically no useful information above the radar horizon (i.e.,beyond the detection limit of the radar sensor) and little or noinformation in the near field (at least at higher altitudes). IR sensorsare typically most effective at shorter ranges (near field), especiallyin weather conditions where the far-field cannot be detected due to thesensor's inability to penetrate obscurants such as rain or fog. Thus,with the example radar and IR sensors, radar image regions above theradar horizon and in the near field can be pre-filtered, and IR imageregions in the far field can be pre-filtered. Other fields and regionsmay be suitable for filtering depending on the sensors, resulting imagesgenerated thereby and the needs of the user or system.

A general spatial filter is illustrated in FIGS. 12A-B. FIG. 12Aillustrates a filter for an image generated by a radar sensor.Specifically, the filter removes information where the radar sensor isleast effective, i.e., above the radar horizon 1200 and in the nearfield 1204, while permitting the remaining radar sensor information 1202to pass and be included in a contrast map. The filtered data isrepresented as darker regions 1200, 1204. Similarly, in FIG. 12B, thefilter removes information where the IR sensor is least effective, i.e.,in the far field 1212, while permitting the remaining information 1210and 1214 to pass and be included in a contrast map. While FIGS. 12A-Bessentially illustrate almost complementary filters, those skilled inthe art will recognize that this will not always be the case withdifferent sensor/image combinations. Different sensors may requiredifferent filter functions.

Once technique for filtering image regions is performed by selectingspace-dependent α and β weighting functions. Continuing with the exampleinvolving radar and IR images, the weighting functions can be selectedto overweight the radar image contribution in those regions where theradar signal is strongest, and, overweight the IR signal everywhereelse.

The weighting function can be implemented through a spatial filter orother smoothing function that does not introduce unnecessary artifacts,e.g., a one-dimensional Gaussian weighting function as follows:α(x,y)=α_(M)e^(−b) ^(M) ^((y−y) ⁰ ⁾+p_(M)β(x,y)=α₁(1−e^(−b) ^(I) ^((y−y) ⁰ ⁾ ² )+p_(I)where:

α_(M) and α_(I) determine the maximum amplitude of the Gaussian function(usually 1, but other values can also be used to overweight one sensor,or to compensate for the pedestal values, P_(M) and P_(I));

b_(M) and b_(I) determine the Gaussian function width, i.e., the regionof interest of the sensor or the region where the sensor information isclustered; and

y₀ shifts the center of the Gaussian function vertically up and down inthe image as required.

More detailed examples of such weighting functions are illustrated inFIGS. 13A-B. FIGS. 13A-B illustrate plots 1300, 1310 of example filtertransparency distributions for respective radar and IR sensors. In eachplot 1300, 1310, the horizontal or “x” axis represents a line orcross-section along the corresponding image. The vertical axis or “y”axis represents filter transparency or transmission capabilities.

Referring to FIG. 13A, the filter plot 1300 illustrates the filterweighting as a function of vertical position in the corresponding FIG.13C. The plot illustrates transmission values, percentages, or ratios:0.0 (no data transmitted), 0.2, 0.4 . . . 1.0 (all data transmitted).Thus, this example filter is designed to de-emphasize the leasteffective portions of the radar image, i.e., above the radar horizon1320 and in the near field 1324. As a result, a filter with a hightransmission ratio (i.e., 1.0) is applied to the most effective portionof the radar image, i.e., in the far field or the middle section of theimage 1322.

Specifically, one example of a radar filter is configured withfull-contrast cycle: 100% transparency at its maximum, in the center ofthe image and 0% at the upper and lower edges of the image. The examplefilter 1300 is constructed with a standard deviation of 50 pixels.Different filter configurations and functions can be utilized dependingon the sensor used and the desired filtering effect.

FIG. 13B illustrates the filtering weighting as a function of verticalposition in the corresponding FIG. 13D. This filter 1310 is designed tode-emphasize the least effective portions of the IR filter, i.e., thecentral image or far-field band, 1332 and emphasize the stronger regions1330, 1334. The example IR filter has 75% maximum contrast: it variesfrom about 25% transparency in the center of the image, to 100% at theupper and lower edges, and has a standard deviation of 50 pixels similarto filter function 1300.

Weighting sensor images in this manner essentially pre-selects imageregions that contain useful and relevant information, and are thereforecandidates for inclusion in the fused image. In addition, by filteringout regions where little information is available, processing time canbe reduced.

The pre-selection or filtering of image regions is further illustratedin FIGS. 14A-F, continuing with the example of radar and IR images.

FIG. 14A illustrates an original radar image 1400 generated by a radarsensor. As can be seen in image 1400, the middle region 1404 or farfield contains the most information compared to regions 1402 (above theradar horizon) and 1406 (near field). FIG. 14B illustrates the filter1410. The filter includes a high transmission section 1414 correspondingto region 1404 of the radar image, and low transmission sections 1412and 1416 corresponding with regions 1402 and 1406 of the radar image.Thus, the filter de-emphasizes regions 1402, 1406 in which radar isleast effective. FIG. 14C illustrates the post-filter radar image 1420in which the farfield or middle region 1404 is emphasized to provide themost relevant information.

Similarly, FIG. 14D illustrates an original IR image 1430 generated byan IR sensor. As can be seen from the image 1430, the top and bottomregions 1432 (above radar horizon) and 1436 (near field) contain themost information compared to region 1434 (far field). FIG. 14Eillustrates a filter 1440. The filter includes high transmission section1442 and 1446 corresponding to regions 1432 and 1436 of the IR image,and low transmission section 1444 corresponding with region 1434 of theIR image. Thus, the filter de-emphasizes region 1434 in which IR isleast effective. FIG. 14F illustrates the post-filter IR image 1450 inwhich the above radar horizon region 1432 and near field region 1436 areemphasized to provide the most relevant information.

For optimal filtering, the weighting function should account for stateor operating parameters depending on the needs and design of thespecific system. For example, as illustrated in FIGS. 15A-E, in the caseof aircraft, filtering can be a function of aircraft roll or othermotions or orientations that result in a rotation of the scene horizon.Thus, filtering can be matched by the orientation of the weightingfunction. Further, filtering can be a function of aircraft pitch andaltitude, both of which affect the effective radar field of view andtypically affect the standard deviation and vertical position of theweighting function.

Thus, for example, FIG. 15A illustrates an original radar image 1500.FIG. 15B illustrates a weighting or filter function 1510 for normalconditions, i.e., without aircraft roll. FIG. 15C illustrates thepost-filter radar image 1520. As a result, both the filter 1510 andfiltered radar image 1520 are parallel to the scene horizon and do notexhibit any angular adjustments.

FIG. 15D illustrates a weighting or filter function 1530 reflecting anaircraft roll of about 5 degrees. More specifically, the transmissiveportion of the filter is rotated about 5 degrees. FIG. 16E illustratesthe post-filter radar image 1540 reflecting the filter function beingrotated about 5 degrees to account for an aircraft roll of about 5degrees.

Combination of Pre-Filtering, Contrast-Based Image Fusion, and LuminanceCorrection

Depending on the sensors and resulting quality of sensor and fusedimages, the spatial pre-filtering and/or luminance correction processescan be applied to images as part of the image fusion processing.

If only contrast-based image fusion and luminance correction areperformed, they will usually be completed in the recited order. If allthree processes are performed, spatial pre-filtering will typically beperformed first, then contrast-based sensor fusion, and finallyluminance correction. These sequences typically result in more effectivefused images while reducing processing time. Luminance correction shouldnormally follow both pre-filtering and contrast-based fusion to mostclosely achieve the desired luminance distribution and to prevent imageluminance distribution from changing as a result of subsequentprocessing. By applying these techniques in this manner, systemperformance is enhanced by minimizing pipeline delays and data latency.These enhancements can be particularly useful in time-intensivesituations that involve the images, e.g., airborne, pilot-in-the-loopapplications, or other applications that use real-time image processing.

Although references have been made in the foregoing description to apreferred embodiment, persons of ordinary skill in the art of designingimage processing systems will recognize that insubstantialmodifications, alterations, and substitutions can be made to thepreferred embodiment described without departing from the invention asclaimed in the accompanying claims.

Thus, while the preferred embodiment is primarily described asprocessing two images from radar and IR sensors in connection with anaircraft, those skilled in the art will recognize that images from othertypes, combinations, and numbers of sensors can be utilized. Forexample, instead of two sensors, the system can be implemented withthree, four, five, or other numbers of sensors. Moreover, instead of aradar and an IR sensor, the system can process images from the same typeof sensors at different wavelengths, ultraviolet (UV) sensors, sensorsbased on an active or passive radio-frequency (RF) system; an ultrasonicsensor, a visible band sensor, e.g., a low-light level visible bandsensor, Charge Coupled Device (CCD), or a color or gray-scale camera.Moreover, persons of ordinary skill in the art will appreciate that thepresent image fusion system and method can be used in other applicationsbesides processing aircraft images. For example, the system and methodcan be used in connection with other moving vehicles, medicalprocedures, surveillance, and other monitoring and image processingapplications involving multiple images or sensors. Additionally, personsof ordinary skill in the art will recognize that a fused or processedimage can be formed based on various selection criteria or processes,greater or maximum contrast values being example criteria.

1. A method of forming a processed image using a plurality of images,each image generated by a respective sensor, comprising: dividing eachimage into a plurality of image regions; generating a contrast map foreach image, each contrast map including a contrast value for each imageregion; applying a selection process to said contrast value forselecting an image region for use in said processed image; and formingsaid processed image with the selected image regions, wherein contrastvalues of contrast maps of respective first, second, and third sensorsare compared together to form said processed image, the method furthercomprising: identifying contrast values from first and second sensorimages to form an intermediate contrast map; wherein applying theselection process comprises applying a selection process to the contrastvalues of the intermediate contrast map and contrast values of acontrast map of a third sensor image.
 2. The method of claim 1, whereindividing the images into the plurality of image regions furthercomprises dividing each image on a pixel-by-pixel basis, into blocks ofpixels, or into arbitrary shaped regions.
 3. The method of claim 1,wherein each sensor detects a different wavelength.
 4. The method ofclaim 1, wherein the plurality of sensors includes an infrared (IR)sensor and a radar sensor.
 5. The method of claim 1, wherein theplurality of sensors includes an infrared (IF) and an ultraviolet (UV)sensor.
 6. The method of claim 1, wherein the plurality of sensorsincludes a radar sensor and an ultraviolet (UV) sensor.
 7. The method ofclaim 1, wherein the plurality of images are generated by two or moreinfrared (IR) sensors, each IR sensor detecting a different wavelength.8. The method of claim 1, wherein applying the selection processincludes comparing competing contrast values of two corresponding imageregions from two respective images.
 9. The method of claim 8, whereinsaid selection process operates to select the greater of the competingcontrast values.
 10. The method of claim 8, wherein comparing competingcontrast values further comprises comparing corresponding contrastvalues of overlapping image regions.
 11. The method of claim 1, whereinthe first and second sensors are infrared (IR) sensors and the thirdsensor is a radar sensor.
 12. The method of claim 1, wherein a sensorimage displays a view from a moving vehicle.
 13. The method of claim 12,wherein the moving vehicle is an aircraft, a watercraft, an automobile,or a train.
 14. The method of claim 1, further comprising adjusting anintensity of one or more regions of said processed image.
 15. The methodof claim 14, further comprising weighting the degree of intensityadjustment.
 16. The method of claim 14, wherein adjusting the intensityfurther comprises adjusting the intensity across said processed image.17. The method of claim 14, wherein adjusting the intensity across saidprocessed image further comprises: selecting one sensor as a referencesensor; determining at least one average intensity value for each regionof the reference sensor image; and adjusting the intensity of one ormore regions in said processed image by combining the determined averageintensity values and intensity values of said processed image.
 18. Themethod of claim 17, wherein the sensors include a radar sensor and aninfrared (IR) sensor, and wherein the reference sensor comprises theradar sensor.
 19. The method of claim 17, wherein the sensors include aradar sensor and an infrared (IR) sensor, and wherein the referencesensor comprises the infrared (IR) sensor.
 20. The method of claim 17,wherein adjusting the intensity of one or more regions in said processedimage further comprises adjusting the intensity of a line in saidprocessed image corresponding to a line in the reference sensor imagefor which the average intensity was determined.
 21. The method of claim20, wherein adjusting the intensity of one or more lines in saidprocessed image further comprises adjusting the intensity of a line insaid processed image that is adjacent to a line in said processed imagecorresponding to the same line in the reference sensor image for whichthe average intensity was determined.
 22. The method of claim 1, beforegenerating the contrast map for each image, further comprising filteringregions of one or more images.
 23. The method of claim 22, whereinfiltering further comprises spatially filtering regions of each image byweighting selected image regions.
 24. The method of claim 23, whereinone sensor comprises a radar sensor, and wherein spatial filtering isperformed by filtering image regions above a radar horizon.
 25. Themethod of claim 1, wherein each of the sensors detects substantially thesame scene such that each of the images is associated with substantiallythe same scene.
 26. The method of claim 25, wherein each of the sensorsdetects a range of wavelengths that is different than a range ofwavelengths detected by the other sensors, such that each of the imagesis associated with a different range of wavelengths.
 27. A method offorming a processed image using a plurality of images, each imagegenerated by a respective sensor, comprising: dividing each image into aplurality of image regions; generating a contrast map for each image,each contrast map including a contrast value for each image region;applying a selection process to said contrast value for selecting animage region for use in said processed image; and forming said processedimage with the selected image regions; wherein generating the contrastmap further comprises performing a convolution to determine the contrastvalue of the contrast map; wherein performing the convolution furthercomprises performing the convolution with a Kernel Kc, wherein[{Kc*S1(x,y), Kc*S2(x,y)}] represents the convolution;$K_{c} = {{\begin{matrix}{- \frac{1}{2\sqrt{2}}} & {- \frac{1}{2}} & {- \frac{1}{2\sqrt{2}}} \\{- \frac{1}{2}} & {2\frac{1 + \sqrt{2}}{\sqrt{2}}} & {- \frac{1}{2}} \\{- \frac{1}{2\sqrt{2}}} & {- \frac{1}{2}} & {- \frac{1}{2\sqrt{2}}}\end{matrix}} \cong {\begin{matrix}{- 0.354} & {- 0.500} & {- 0.354} \\{- 0.500} & 3.414 & {- 0.500} \\{- 0.354} & {- 0.500} & {- 0.354}\end{matrix}}}$ S1 represents image regions of a first image; S2represents image regions of a second image; and (x,y) represent spatialcoordinates of the images.
 28. The method of claim 27, wherein dividingthe images into the plurality of image regions further comprisesdividing each image on a pixel-by-pixel basis, into blocks of pixels, orinto arbitrary shaped regions.
 29. The method of claim 27, wherein eachsensor detects a different wavelength.
 30. The method of claim 27,wherein applying the selection process includes comparing competingcontrast values of two corresponding image regions from two respectiveimages.
 31. The method of claim 30, wherein said selection processoperates to select the greater of the competing contrast values.
 32. Themethod of claim 30, wherein comparing competing contrast values furthercomprises comparing corresponding contrast values of overlapping imageregions.
 33. A system for combining a plurality of images to form afinal image, comprising: a plurality of sensors that generate respectiveimages; a processor configured to divide each image into a plurality ofimage regions, generate a contrast map for each image, each contrast mapincluding a contrast value for each image region, apply a selectioncriterion to said contrast value for selecting an image region for usein said processed image, and form said processed image with the selectedimage regions; wherein contrast values of contrast maps of a respectivefirst, second, and third sensors are compared together to form the finalimage; wherein the processor is further configured to identify contrastvalues from first and second sensor images to form an intermediatecontrast map; and wherein the processor applies the selection criterionby applying a selection process to the contrast values of theintermediate contrast map and contrast values of a contrast map of athird sensor image.
 34. The system of claim 33, wherein the processor isconfigured to divide each image into individual pixels, into blocks ofpixels, or into arbitrary shaped regions.
 35. The system of claim 33,wherein each sensor detects a different wavelength.
 36. The system ofclaim 33, wherein the plurality of sensors includes an infrared (IR)sensor and a radar sensor.
 37. The system of claim 33, wherein theplurality of sensors includes an infrared (IR) and an ultraviolet (UV)sensor.
 38. The system of claim 33, wherein the plurality of sensorsincludes a radar sensor and an ultraviolet (UV) sensor.
 39. The systemof claim 33, wherein the plurality of sensors includes two or moreinfrared (IR) sensors, each IR sensor detecting a different wavelength.40. The system of claim 33, wherein the processor is further configuredto compare competing contrast values of two corresponding image regionsfrom two respective images.
 41. The system of claim 40, wherein theprocessor is further configured to select the greater of the competingcontrast values.
 42. The system of claim 40, wherein the processor isconfigured to compare corresponding contrast values of overlapping imageregions.
 43. The system of claim 33, wherein the first and secondsensors are infrared (IR) sensors and the third sensor is a radarsensor.
 44. The system of claim 33, wherein a sensor image displays aview from a moving vehicle.
 45. The system of claim 44, wherein themoving vehicle comprises an aircraft, a watercraft, an automobile, or atrain.
 46. The system of claim 33, wherein the processor is furtherconfigured to adjust an intensity of one or more regions of saidprocessed image.
 47. The system of claim 46, wherein the processor isconfigured to adjust the intensity across said processed image.
 48. Thesystem of claim 47, wherein the processor is configured to weight thedegree of intensity adjustment.
 49. The system of claim 47, wherein theprocessor is further configured to select one sensor as a referencesensor, determine at least one average intensity value for each regionof the reference sensor image across the reference sensor image, andadjust the intensity of one or more regions in said processed image bycombining the determined average intensity values and intensity valuesof said processed image.
 50. The system of claim 49, wherein the sensorsinclude a radar sensor and an infrared (IR) sensor, and wherein thereference sensor comprises the radar sensor.
 51. The system of claim 49,wherein the sensors include a radar sensor and an infrared (IR) sensor,and wherein the reference sensor comprises the infrared (IR) sensor. 52.The system of claim 49, wherein the processor is configured to adjustthe intensity of a line in said processed image corresponding to a linein the reference sensor image for which the average intensity wasdetermined.
 53. The system of claim 49, wherein the processor isconfigured to adjust the intensity of a line in said processed imagethat is adjacent to a line in said processed image corresponding to thesame line in the reference sensor image for which the average intensitywas determined.
 54. The system of claim 33, wherein the processor isconfigured to filter one or more image regions.
 55. The system of claim54, wherein the processor is configured to filter the one or more imageregions by weighting selected image regions.
 56. The system of claim 54,wherein one sensor comprises a radar sensor, and wherein the processoris further configured to spatially filter image regions above a radarhorizon.
 57. The system of claim 33, wherein each of the sensors detectssubstantially the same scene such that each of the respective images isassociated with substantially the same scene.
 58. The system of claim33, wherein each of the sensors detects a range of wavelengths that isdifferent than a range of wavelengths detected the other sensors, suchthat each of the respective images is associated with a different rangeof wavelengths.
 59. A system for combining a plurality of images to forma final image, comprising: a plurality of sensors that generaterespective images; and a processor configured to divide each image intoa plurality of image regions, generate a contrast map for each image,each contrast map including a contrast value for each image region,apply a selection criterion to said contrast value for selecting animage region for use in said processed image, and form said processedimage with the selected image regions; wherein the processor isconfigured to generate the contrast map by performing a convolution todetermine the contrast value of the contrast map; and wherein theprocessor is configured to perform the convolution with a Kernel Kc,wherein [{Kc*S1(x,y), Kc*S2(x,y)}] represents the convolution;$K_{c} = {{\begin{matrix}{- \frac{1}{2\sqrt{2}}} & {- \frac{1}{2}} & {- \frac{1}{2\sqrt{2}}} \\{- \frac{1}{2}} & {2\frac{1 + \sqrt{2}}{\sqrt{2}}} & {- \frac{1}{2}} \\{- \frac{1}{2\sqrt{2}}} & {- \frac{1}{2}} & {- \frac{1}{2\sqrt{2}}}\end{matrix}} \cong {\begin{matrix}{- 0.354} & {- 0.500} & {- 0.354} \\{- 0.500} & 3.414 & {- 0.500} \\{- 0.354} & {- 0.500} & {- 0.354}\end{matrix}}}$ S1 represents image regions of a first image; S2represents image regions of a second image; and (x,y) represent spatialcoordinates of the images.
 60. The system of claim 59, wherein theprocessor is configured to divide each image into individual pixels,into blocks of pixels, or into arbitrary shaped regions.
 61. The systemof claim 59, wherein each sensor detects a different wavelength.
 62. Thesystem of claim 59, wherein the processor is further configured tocompare competing contrast values of two corresponding image regionsfrom two respective images.
 63. The system of claim 62, wherein theprocessor is further configured to select the greater of the competingcontrast values.
 64. The system of claim 62, wherein the processor isconfigured to compare corresponding contrast values of overlapping imageregions.
 65. A method of forming a processed image using a plurality ofimages, each image generated by a respective sensor, comprising:dividing each image into a plurality of image regions; generating acontrast map for each image, each contrast map including a contrastvalue for each image region; applying a selection process to saidcontrast value for selecting an image region for use in said processedimage; and forming said processed image with the selected image regionswherein dividing the images into the plurality of image regions furthercomprises dividing each image on a pixel-by-pixel basis, into blocks ofpixels, or into arbitrary shaped regions, and wherein applying theselection process includes comparing competing contrast values ofcorresponding image regions from respective images.
 66. The method ofclaim 65, wherein: each sensor detects a range of wavelengths that isdifferent than a range of wavelengths detected by the other sensors suchthat each of the images is associated with a different range ofwavelengths; dividing each image comprises defining sets ofcorresponding image regions by dividing each image into a plurality ofimage regions each of which is substantially aligned with andsubstantially overlaps with one image region in each of the otherimages; and selecting an image region for use in the processed imagecomprises comparing, for each set of corresponding image regions, thecontrast values for each image region in the set and selecting one imageregion from the set based on the comparing.
 67. The method of claim 65,wherein the sensors are fixed to a moving vehicle, wherein each sensordetects substantially the same field of view from the moving vehiclesuch that the images represent substantially the same field of view. 68.The method of claim 65, wherein the plurality of sensors includes atleast two of an infrared sensor, an ultraviolet sensor, and a radiofrequency sensor.
 69. The method of claim 68, wherein the radiofrequency sensor comprises an imaging radar sensor.
 70. The method ofclaim 65, wherein each sensor is one of an infrared sensor, anultraviolet sensor, or a radio frequency sensor.
 71. The method of claim65, wherein each sensor detects a range of wavelengths that is differentthan a range of wavelengths detected by the other sensors.
 72. Themethod of claim 65, comprising adjusting an intensity value associatedwith at least one region of the processed image based on an averageintensity value associated with at least a portion of one of theprocessed image.
 73. The method of claim 72, wherein adjusting theintensity value further comprises adjusting the intensity value acrosssaid processed image.
 74. The method of claim 73, wherein adjusting theintensity value across said processed image further comprises: selectingone sensor as a reference sensor; determining an average intensity valueof each region of the reference sensor image; and adjusting theintensity value of one or more regions in said processed image bycombining the determined average intensity values and intensity valuesof said processed image.
 75. The method of claim 74, wherein adjustingthe intensity value of one or more regions in said processed imagefurther comprises adjusting the intensity value of a line in saidprocessed image corresponding to a line in the reference sensor imagefor which the average intensity value was determined.
 76. The method ofclaim 75, wherein adjusting the intensity value of one or more lines insaid processed image further comprises adjusting the intensity value ofa line in said processed image that is adjacent to a line in saidprocessed image corresponding to the same line in the reference sensorimage for which the average intensity value was determined.
 77. Themethod of claim 74, wherein a scene horizon is repositioned at an anglerelative to an image horizon, further comprising: determining an averageintensity value of the reference sensor image on a pixel-by-pixel basis;and adjusting the intensity value of said processed image on apixel-by-pixel basis.
 78. The method of claim 77, wherein the scenehorizon is repositioned due to roll, bank, yaw or pitch motions.
 79. Themethod of claim 65, comprising filtering regions of one or more imagesbefore generating the contrast map for each image.
 80. The method ofclaim 79, wherein filtering comprises spatially filtering regions ofeach image by weighting selected image regions.
 81. The method of claim80, wherein one sensor comprises a radar sensor, and wherein spatialfiltering is performed by filtering image regions above a radar horizon.82. The method of claim 65, wherein generating the contrast map furthercomprises performing a convolution to determine the contrast value ofthe contrast map.
 83. The method of claim 65, wherein said selectionprocess operates to select the greater of the competing contrast values.84. The method of claim 65, wherein comparing competing contrast valuesfurther comprises comparing corresponding contrast values of overlappingimage regions.
 85. The method of claim 65, wherein a sensor imagedisplays a view from a moving vehicle.
 86. The method of claim 85,wherein the moving vehicle is an aircraft, a watercraft, an automobile,or a train.
 87. A method of forming a processed image using a pluralityof images, each image generated by a respective sensor to form aprocessed image, comprising: filtering portions of one or more images;comparing contrast values of the images by dividing each image into aplurality of image regions, generating a contrast map for each image,each contrast map including contrast values for each image region ofeach image; comparing contrast values in each contrast map of the imageregions, identifying maximum contrast values based on the comparison ofcontrast values, and selecting image regions corresponding to themaximum contrast values, forming the processed image with the selectedimage regions; and adjusting an intensity of one or more portions of theprocessed image by selecting one sensor as a reference sensor,determining at least one average intensity value for one or more regionsof the reference sensor image across the reference sensor image, andadjusting the intensity of one or more regions in the processed image bycombining the determined average intensity values and intensity valuesof the processed image; wherein generating the contrast map furthercomprises performing a convolution to determine the contrast value ofthe contrast map.
 88. The method of claim 87, wherein performing theconvolution comprises performing a convolution over two image regionsthat are spatially pre-registered or aligned.
 89. The method of claim87, wherein performing the convolution comprises using a kernelincluding values that reflect a distance metric from a center position.90. The method of claim 87, wherein the plurality of sensors includes atleast two of an infrared sensor, an ultraviolet sensor, and a radiofrequency sensor.
 91. The method of claim 90, wherein the radiofrequency sensor comprises an imaging radar sensor.
 92. The method ofclaim 87, wherein each sensor detects a range of wavelengths that isdifferent than a range of wavelengths detected by the other sensors. 93.The method of claim 87, wherein applying the selection process includescomparing competing contrast values of two corresponding image regionsfrom two respective images.
 94. The method of claim 87, wherein eachsensor is one of an infrared sensor, an ultraviolet sensor, or a radiofrequency sensor.
 95. The method of claim 94, wherein the radiofrequency sensor comprises an imaging radar sensor.
 96. The method ofclaim 87, wherein each sensor detects a range of wavelengths that isdifferent than a range of wavelengths detected by the other sensors. 97.The method of claim 87, wherein selecting image regions includescomparing competing contrast values of two corresponding image regionsfrom two respective images.
 98. The method of claim 87, whereinfiltering comprises spatially filtering regions of each image byweighting selected image regions.
 99. The method of claim 98, whereinone sensor comprises a radar sensor, and wherein spatial filtering isperformed by filtering image regions above a radar horizon.
 100. Amethod of forming a processed image using a plurality of images, eachimage generated by a respective sensor, comprising: comparing contrastvalues of contrast maps of the images by defining a plurality of sets ofcorresponding image regions in the plurality of images; generatingcontrast maps with contrast values for the sets of corresponding imageregions in the plurality of images; identifying, for each set ofcorresponding image regions, one contrast value as a maximum contrastvalue, and selecting image regions corresponding to the maximum contrastvalues, forming a processed image using the selected image regions; andadjusting an intensity of at least one portion of the processed imageby: determining at least one intensity value for the at least oneportion of the processed image, selecting one of the sensors as areference sensor, determining an average intensity value for one or moreregions of an image generated by the reference sensor, and adjusting theat least one intensity value for the at least one portion of theprocessed image in accordance with the determined average intensityvalue.
 101. A system for combining a plurality of images to form a finalimage, comprising: a plurality of sensors that generate respectiveimages; a processor configured to divide each image into a plurality ofimage regions, generate a contrast map for each image, each contrast mapincluding a contrast value for each image region, apply a selectioncriterion to said contrast value for selecting an image region for usein said processed image, and form said processed image with the selectedimage regions; wherein the processor is further configured to divide theimages into the plurality of image regions by dividing each image on apixel-by-pixel basis, into blocks of pixels, or into arbitrary shapedregions, and wherein the process is further configured to apply theselection process by comparing competing contrast values ofcorresponding image regions from respective images.
 102. The system ofclaim 101, wherein each sensor detects a different wavelength.
 103. Thesystem of claim 101, wherein at least one sensor is of the followinggroup: an infrared (IR) sensor, a radar sensor, and an ultraviolet (UV)sensor.
 104. The system of claim 101, wherein the processor is furtherconfigured to compare competing contrast values of two correspondingimage regions from two respective images.
 105. The system of claim 101,wherein the processor is further configured to select the greater of thecompeting contrast values.
 106. The system of claim 101, wherein theprocessor is configured to compare corresponding contrast values ofoverlapping image regions.
 107. The system of claim 101, wherein asensor image displays a view from a moving vehicle.
 108. The system ofclaim 107, wherein the moving vehicle comprises an aircraft, awatercraft, an automobile, or a train.
 109. The system of claim 101,wherein the processor is further configured to adjust an intensity ofone or more regions of said processed image.
 110. The system of claim109, wherein the processor is configured to adjust the intensity acrosssaid processed image.
 111. The system of claim 110, wherein theprocessor is configured to weight the degree of intensity adjustment.112. The system of claim 110, wherein the processor is furtherconfigured to select one sensor as a reference sensor, determine atleast one average intensity value for each region of the referencesensor image across the reference sensor image, and adjust the intensityof one or more regions in said processed image by combining thedetermined average intensity values and intensity values of saidprocessed image.
 113. The system of claim 112, wherein the processor isconfigured to adjust the intensity of a line in said processed imagecorresponding to a line in the reference sensor image for which theaverage intensity was determined.
 114. The system of claim 112, whereinthe processor is configured to adjust the intensity of a line in saidprocessed image that is adjacent to a line in said processed imagecorresponding to the same line in the reference sensor image for whichthe average intensity was determined.
 115. The system of claim 101,wherein the processor is configured to filter one or more image regions.116. The system of claim 115, wherein the processor is configured tofilter the one or more image regions by weighting selected imageregions.
 117. The system of claim 115, wherein one sensor comprises aradar sensor, and wherein the processor is further configured tospatially filter image regions above a radar horizon.
 118. The system ofclaim 101, wherein the processor is configured to generate the contrastmap by performing a convolution to determine the contrast value of thecontrast map.
 119. A system for forming a processed image using aplurality of images, comprising: a first sensor that generates a firstimage; a second sensor that generates a second image, wherein the firstand second images are divided into a plurality of image regions; aprocessor configured to filter one or more portions of one or moreimages; compare contrast values of the images by dividing each imageinto a plurality of image regions, generating a contrast map for eachimage, each contrast map including contrast values for each image regionof each image, comparing contrast values in each contrast map of theimage regions, identifying maximum contrast values, selecting imageregions corresponding to the maximum contrast values, and forming theprocessed image with the selected image regions; and adjust an intensityof one or more regions of the final image by selecting one sensor as areference sensor, determining at least one average intensity value foreach region of the reference sensor image across the reference sensorimage, and adjusting the intensity of one or more regions in saidprocessed image by combining the determined average intensity values andintensity values of the final image; wherein the processor is furtherconfigured to divide the images into the plurality of image regions bydividing each image on a pixel-by-pixel basis, into blocks of pixels, orinto arbitrary shaped regions, and wherein the processor is furtherconfigured to apply the selection process by comparing competingcontrast values of corresponding image regions from respective images.